1. Field of the Invention
The present invention relates to an image processing apparatus for generating a color signal for each pixel using data interpolation processing of color signals for a target pixel and at least one pixel within the vicinity thereof the target pixel, based on color signals output from a solid state image pickup device. The solid state image pickup device includes a plurality of pixel sections on which a plurality types of color filters are separately arranged. The present invention further relates to an imaging apparatus including the image processing apparatus, such as a single-chip digital camera, a video camera and a mobile phone with a camera. The present invention further relates to a an image processing method corresponding to the image processing apparatus, an image processing program for having a computer execute the image processing method, and a computer-readable recording medium on which the image processing program is recorded.
2. Description of the Related Art
Conventionally, only information on the brightness of light is output from a solid state image pickup device such as a CCD and a CMOS used in a digital camera or a video camera.
In a single-chip camera having one solid state image pickup device, in order to obtain color information, the front of the CCD is covered with color filters which respectively pass only one color component for each pixel included in the CCD, and each of the pixel sections outputs a corresponding color component when in operation. For example, when primary-color filters are used, there are three types of color components respectively for the three primary colors, namely, R (red), G (green), and B (blue).
A number of suggestions have been made as to how to arrange the color components of the color filters.
A typical example is a Bayer array shown in FIG. 11. In the Bayer array, “G”s which make relatively larger contribution to luminance signals are arranged in a checkerboard pattern, and “R”s and “B”s are arranged in the rest of the area, also in a checkerboard pattern.
At a point in time when a signal is output from a CCD covered with color filters arranged in a Bayer array or the like, only color information for one color out of the RGB color components is obtained for each pixel.
Thus, interpolation processing is performed by an image processing section in order to obtain other color information, by calculating for each pixel information for the other two colors, using color signal values of the pixels in the vicinity of the target pixel on the basis of conjecture.
In addition, interpolation processing is sometimes performed on the color information of one color for each pixel that is obtained by a CCD, using the color signal value of the target pixel itself and the color signal values of the pixels in the vicinity of the target pixel (which may be referred to as “vicinity pixels”).
The following describes the bilinear method, which is a conventional interpolation processing method, with reference to FIGS. 12A and 12B.
FIG. 12A shows a Bayer array in a case where the color component of a target pixel is G.
FIG. 12B shows a Bayer array in a case where the color component of a target pixel is either R or B. It should be noted that a pair made up of P and Q represents the color components R and B, or the color components B and R. In other words, when P represents R, Q represents B. On the other hand, when P represents B, Q represents R.
When color filters are arranged in a Bayer array, interpolation formulas <Formulae 1> according to the bilinear method can be expressed as below:
<Formulae 1>
In the case where a target pixel is G33 as shown in FIG. 12A, (the color element of the target pixel being G), <Formulae 1> is as below:Go=G33 or Go=(G33×4+G22+G42+G24+G44)/8Po=(P32+P34)/2Qo=(Q23+Q43)/2
In the case where a target pixel is P33 as shown in FIG. 12B, (the color element of the target pixel being either R or B), <Formulae 1> is as below:Go=(G32+G23+G43+G34)/4Po=P33Qo=(Q22+Q42+Q24+Q44)/4where                Go denotes an output G signal of the target pixel,        Po denotes an output P signal of the target pixel,        Qo denotes an output Q signal of the target pixel, and        (P, Q)=(R, B) or (B, R).        
However, in a case where interpolation processing is performed on color signals for an image changing from white to black, using the conventional bilinear method shown above, it is known that a false color signal (hereafter simply referred to as a moiré signal) may be generated at an edge portion.
As a method for reducing such moiré signals, Japanese Patent No. 2931520 proposes an interpolation processing method by which occurrence of moiré signals is restrained with the use of color correlations.
An image in general has characteristics such as “in a local area with color signals, the ratio of a low-frequency component to a high-frequency component of a specific color component is equal to a ratio in another color component”, and also “in a local area with color signals, the difference between a G signal and a R signal or between a G signal and a B signal is approximately equal to each other between adjacent pixels.”
The interpolation processing method disclosed in Japanese Patent No. 2931520 utilizes the characteristics mentioned above and detects correlations in the color signals arranged in a Bayer array between the vertical direction (the lengthwise direction) and the horizontal direction (the widthwise direction) so that the interpolation processing is performed with the use of the correlations. Specifically, color signals by an interpolation processing for a cage where there is a correlation in a vertical direction and color signals by an interpolation processing form a ase where there is a correlation in a horizontal direction are calculated in advance, and these color signals calculated by the interpolation processing for different directions are mixed in accordance with the strengths of the vertical correlation and the horizontal correlation.
According to the conventional interpolation processing method disclosed in Japanese Patent No, 2931520 mentioned above, the moiré signals are reduced compared to the case where the bilinear method is used. However, an image in general has a large number of edges in various directions other than the vertical direction (the lengthwise direction) and the horizontal (the widthwise direction) direction. In accordance with the interpolation processing method disclosed in Japanese Patent No. 2931520, color signals obtained from interpolation processing in the two directions are mixed based on two types of correlation data (the strength levels of correlations) for the vertical direction and the horizontal direction, even though the large number of edge directions exist; therefore, in some cases, moiré signals may occur due to an edge in a diagonal direction.
For example, the following describes a case where a solid state image pickup device such an a CCD includes color filters as shown in FIG. 13D so that monochrome color signals shown in FIG. 13A are obtained as a result of an A/D conversion.
FIG. 13A corresponds to a part of an edge portion that is at an angle towards the top right from a vertical direction. In the cave where interpolation processing is performed on the color signal values in FIG. 13A, according to the correlations using color differences for five pixels by three pixels, as disclosed in Japanese Patent No. 2931520, the color signals are expressed as follower
                    Rv        =                ⁢                              G            ⁢                                                  ⁢            33                    +                                    (                                                R                  ⁢                                                                          ⁢                  32                                +                                  R                  ⁢                                                                          ⁢                  34                                            )                        /            2                    -                                    (                                                G                  ⁢                                                                          ⁢                  31                                +                                  2                  ×                  G                  ⁢                                                                          ⁢                  33                                +                                  G                  ⁢                                                                          ⁢                  35                                            )                        /            4                                                  =                ⁢        64                                Gv        =                ⁢                  G          ⁢                                          ⁢          33                                        =                ⁢        2                                Bv        =                ⁢                              G            ⁢                                                  ⁢            33                    +                      B            ⁢                                                  ⁢            23                    -                      G            ⁢                                                  ⁢            23                                                  =                ⁢                              G            ⁢                                                  ⁢            33                    +                      B            ⁢                                                  ⁢            23                    -                      {                                          B                ⁢                                                                  ⁢                23                            +                                                (                                                            G                      ⁢                                                                                          ⁢                      22                                        +                                          G                      ⁢                                                                                          ⁢                      24                                                        )                                /                2                            -                                                                      ⁢                              (                                          B                ⁢                                                                  ⁢                21                            +                              2                ×                B                ⁢                                                                  ⁢                23                            +                              B                ⁢                                                                  ⁢                25                                      )                    /          4                }                                =                ⁢                  -          60                                        Rh        =                ⁢                              G            ⁢                                                  ⁢            33                    +                      R            ⁢                                                  ⁢            32                    -                      G            ⁢                                                  ⁢            32                                                  =                ⁢                              G            ⁢                                                  ⁢            33                    +                      R            ⁢                                                  ⁢            32                    -                      {                                          R                ⁢                                                                  ⁢                32                            +                                                (                                                            G                      ⁢                                                                                          ⁢                      22                                        +                                          G                      ⁢                                                                                          ⁢                      42                                                        )                                /                2                            -                                                                      ⁢                              (                                          R                ⁢                                                                  ⁢                12                            +                              2                ×                R                ⁢                                                                  ⁢                32                            +                              R                ⁢                                                                  ⁢                52                                      )                    /          4                }                                =                ⁢        64                                Gh        =                ⁢                  G          ⁢                                          ⁢          33                                        =                ⁢        2                                Bh        =                ⁢                              G            ⁢                                                  ⁢            33                    +                                    (                                                B                  ⁢                                                                          ⁢                  23                                +                                  B                  ⁢                                                                          ⁢                  43                                            )                        /            2                    -                                    (                                                G                  ⁢                                                                          ⁢                  13                                +                                  2                  ×                  G                  ⁢                                                                          ⁢                  33                                +                                  G                  ⁢                                                                          ⁢                  53                                            )                        /            4                                                  =                ⁢        64            
It should be noted, however, that the color signals Rv, Gv, and Bv are color signals obtained when a processing that is suitable for a case where the correlation in the vertical direction is strong is performed. On the other hand, the color signals Rh, Gh, and Bh are color signals obtained when a processing that is suitable for a case where the correlation in the horizontal direction is strong is performed.
When the calculation formulae for correlation coefficients with directions disclosed in Japanese Patent No. 2931520 are used, the correlation coefficient for the horizontal direction is zero; therefore, when a color signal is to be calculated, only the color signals in the vertical direction are used as references.
Accordingly.Ro=Rv=64Go=Gv=2 andBo=Bv=−60 are obtained.
Originally, in a monochrome image, R=G=B should be satisfied; however, the calculated color signals have relationship such as Ro≠Go≠Bo. This means that moiré signals are generated.
The following describes this in further detail.
In the interpolation processing using color correlations disclosed in Japanese Patent No. 2931520, the color components other than G for the target pixel are calculated according to the correlation between the G component and each of the color components other than G, making use of the characteristic that a G component is always, included in the vertical direction and the horizontal direction of the color filters arranged in a Bayer array. For example, the following is an explanation in which the horizontal direction in FIG. 12A is used as an example.
As shown in FIG. 12A, the horizontal line that includes the target pixel G33 contains Gs and Qs. Accordingly, it is possible to calculate Q33, which is the Q component of the target pixel, using these values of Gs and Qs. In addition, another horizontal line that is positioned right above the horizontal line and includes P32 contains Gs and Ps. Accordingly, it is possible to calculate G32, which is the G component of the target pixel, using these values of Gs and Ps.
According to the interpolation processing method disclosed in Japanese Patent No. 2931520, as mentioned above, focus is given to the pixels P32 and G33 shown in FIG. 12A, P33 is calculated with the use of either the characteristic that the differences between pairs of a G and a P in the local are substantially equal to each other,    i.e. “P33=G33=P32−G32”, orthe characteristic that the ratios in pairs of a P and a G in the local area are substantially equal to each other,    i.e. “P33/G33=P32/G32”.
Since an image in general has edges in various directions, it is preferable, in terms of detecting directions, to detect a larger number of directions than only two directions, namely the vertical direction and the horizontal direction.
According to the interpolation processing method disclosed in Japanese Patent No. 2931520, however, it is not possible to perform an interpolation processing while the color correlations in the diagonal directions are taken into account.
The reason for this is because, for example in FIG. 12A, there are only G components diagonally at the angle of 45 degrees, and, for example in FIG. 128, there are only P components and Q components (B components and R components) diagonally at the angle of 45 degrees.
The present invention aims to solve the problem in the conventional technique and to provide an image processing apparatus, an imaging apparatus including the image processing apparatus, an image processing method corresponding to the image processing apparatus, an image processing program for having a computer execute the image processing method, and a computer-readable recording medium on which the image processing program is recorded. Bach of which makes it possible to further restrain occurrence of moiré signals with the use of correlations between color signals with respect to edges in a plurality of directions including a diagonal direction, when all color signals for each pixel are generated from color signals output from a solid state image pickup device in which color filters for a plurality of colors are separately arranged in each pixel section, for example, in a Bayer array, through an interpolation processing using color signals for the target pixel and a pixel in the vicinity thereof.